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On  November 


7,  1979  the  New  York  Times  carried  a  front  page  story  headlined 
"SOVIET  DISCOVERY  kCCKS  WORLD  OF  MATHEMATICS" 


According  to  the  story,  which  had  been  picked  up  from  the  magazine  Science, 
"apart  from  its  profound  theoretical  interest,  the  discovery  may  be  applicable 
in  weather  prediction,  complicated  industrial  processes,  petroleum  refining,  the 
scheduling  of  workers  at  large  factories,  secret  codes  and  many  other  things". 
The  New  York  Times  coverage  generated  substantial  controversy  concerning  the 
merits  of  the  discovery,  a  new  algorithm  for  doing  linear  programming.  Near  the 
end  of  this  article  we  will  evaluate  this  algorithm. 


The  New  York  Times  story  concerned  the  "computational  complexity"  of  the 
linear  programming  problem.  Although  computational  complexity  does  not  always 
make  front  page  news,  it  is  a  very  active  and  important  research  area.  Its  subject 
is  the  determination  of  the  intrinsic  difficulty  of  mathematically  posed  problems 
arising  in  many  disciplines.  The  study  of  complexity  has  led  to  more  efficient 
algorithms  than  those  previously  known  or  suspected.  We  illustrate  some  of  the 
important  ideas  of  computational  complexity  with  the  example  of  matrix  multipli- 
cat ion . 
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COMPUTATIONAL  COMPLEXITY  OF  MATRIX  MULTIPLICATION.  We  consider  first  the 
multiplication  of  2x2  matrices.  T„t 


ran  ai2 


la21  a22 


/C11  C12 


B 


'11  b12 
b21  b22 


^C21  C22 
Given  A,  B,  we  seek  C  *  AB  . 


The  classical  algorithm  computes  C  by 
C11  *  all  bll  +  a12  b21’  GtC' 

at  a  cost  of  eight  multiplications. 

Until  the  late  sixties  no  one  'seems  to  have  been  asked  whether  two  matrices  could 
be  multiplied  in  less  than  eight  scalar  multiplications.  Then  Strassen  showed  that  7 
scalar  multiplications  are  sufficient. 

Consider  next  the  multiplication  of  N  x  N  matrices.  The  classical  algorithm  uses 
3 

N  arithmetic  operations.  (In  this  article  we  disregard  constants  in  giving  the  cost 
of  our  algorithm.)  By  repeated  partitioning  of  N  by  N  matrices  into  2  by  2  sub¬ 
matrices,  two  matrices  can  be  multiplied  in  N^°^2^  n,  ^  arithmetic  operations. 

After  a  decade  during  which  there  was  practically  no  progress  on  decreasing  the 

number  of  arithmetic  operations  used  in  matrix  multiplication,  Schonhage  and  Pan 
? . 52 

showed  N~  arithmetic  operations  are  sufficient.  The  number  2.52  is  just  the 
current  state  of  our  knowledge  and  researchers  expect  the  exponent  will  be  further 
decreased . 

We  must  emphasize  that  the  above  results  are  of  theoretical  rather  than  practical 
value.  The  value  of  N  has  to  be  enormous  before  the  new  algorithm  would  be  faster 
than  the  classical  one.  On  the  other  hand,  there  are  some  problems  for  which  new 
algorithms  have  had  profound  influence.  A  good  example  is  provided  by  the  finite 

Fourier  transform  on  N  points.  The  fast  Fourier  transform  uses  only  N  log  N  arithmetic 

2 

operations  compared  to  N  for  the  classical  algorithms.  Since  N  log  N  is  much  smaller  : 
than  for  even  moderate  values  of  N  and  since  the  finite  Fourier  tranform  is  often 
needed  for  a  large  number  of  points,  the  introduction  of  the  fast  Fourier  transform 

has  revolutionized  a  number  of  scientific  fields.  , 
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Using  the  matrix  multiplication  example  we  introduce  some  basic  terminology. 


The  minimal  number  of  arithmetic  operations  is  called  the  computational  complexity 
(or  problem  complexity)  of  the  matrix  multiplication  problem.  We  often  write  complexity 
for  brevity. 

2  52 

The  complexity  of  matrix  multiplication  is  unknown.  An  upper  bound  is  N 
2 

A  lower  bound  is  N  .  Since  this  lower  bound  is  linear  in  the  number  of  inputs  and  out¬ 
puts  we  say  it  is  trivial  lower  bound.  No  non-trivial  lower  bound  is  known. 

Algorithm  complexity  is  the  cost  of  a  particular  algorithm.  This  should  be  con¬ 
trasted  with  problem  complexity  which  is  the  minimal  cost  over  all  possible  algorithms. 
People  who  do  not  work  in  complexity  theory  sometimes  confuse  these  two  terms. 

Fast  Algorithm  is  a  qualitative  term  meaning  faster  than  a  classical  algorithm 
or  faster  than  previously  known  algorithms.  An  optimal  algorithm  is  one  whose  com¬ 
plexity  equals  the  problem  complexity. 

Table  1  summarizes  the  present  state  of  our  knowledge  concerning  matrix  multi¬ 
plication. 


SUMMARY  OF  MATRIX  MULTIPLICATION 


upper  bound 

lower  bound 


complexity 
optimal  algorithm 
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2.52 
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unknown 

unknown 


TABLE  1 


3 


COMPUTATIONAL  COMPLEXITY  IN  GENERAL.  We  define  a  model  of  computation  stating 
which  "operations"  or  "steps"  are  permissible  and  how  much  they  cost.  In  the  model 
we  can  ask  tlu  same  questions  as  in  the  matrix  multiplication  example.  For  instance 
we  seek: 

problem  complexity 
upper  bounds 
lower  bounds 
fast  algorithms 
optimal  algorithms 

Typically,  an  upper  bound  is  the  cost  of  an  algorithm  for  solving  the  problem. 

A  lower  bound  must  be  established  thru  a  theorem  that  states  there  does  not  exist  an 
algorithm  whose  cost  is  less  than  the  lower  bound.  Not  surprisingly,  lower  bounds 
are  far  harder  to  establish  than  upper  bounds.  (An  exception  is  provided  by  analytic 
computational  complexity  where  a  very  powerful  adversary  principle  provides  lower 
bounds.  See  discussion  below.) 


SOME  MODELS  OF  COMPUTATION.  Numerous  models  of  computation  have  been  studied. 

In  our  matrix  multiplication  example  we  counted  arithmetic  operations.  In  the  study 
of  combinatorial  problems  we  count  comparisons.  Very  significant  results  have  been 
obtained  for  space  and  time  complexity  in  a  Turing  Machine  model.  Another  Important 
model  is  a  random  access  machine  (RAM).  Other  models  are  appropriate  for  studying 
parallel,  asynchronous,  or  VLSI  computation. 

Often  we  assign  a  "size"  N  to  a  problem.  If  the  number  of  operations  or  steps 
required  to  solve  a  problem  is  an  exponential  function  of  N  we  say  the  problem  has 
exponential  time  complexity.  If  the  problem  requires  a  number  of  operations  which 
is  a  polynomial  function  of  N  we  say  the  problem  has  polynomial  time  complexity. 

An  elegant  theory  for  an  abstract  complexity  model  is  based  on  two  axioms.  One 
result  in  this  abstract  setting  is  the  famous  "Speed-up  Theorem"  which  implies  there 
cannot  be  a  fastest  algorithm  for  a  computable  function. 
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TYPICAL  APPLICATIONS  OF  COMPUTATIONAL  COMPLEXITY.  The  complexity  of  numerous  pro 
blems  has  been  studied.  To  illustrate  the  variety  of  problems  we  exhibit  a  dozen 
drawn  from  various  areas. 

1.  Compute  the  finite  Fourier  transform  at  N  points. 

2.  Determine  if  an  N  digit  integer  is  prime;  if  not  determine  the  factors. 

3.  Solve  an  elliptic  partial  differential  equation  to  within  e 

4.  Compute  the  Kendall  rank  correlation  at  N  points. 

3.  Generate  a  funtion  to  error  less  than  e  from  values  of  the  function  at  N 
points. 

6.  Multiply  two  polynomials  of  degree  N. 

7.  Prove  all  theorems  of  length  less  than  N  in  a  certain  axiom  system. 

8.  Solve  the  traveling  salesman  problem  on  N  cities. 

9.  Solve  to  within  L  a  large  sparse  linear  system  of  order  N  whose  matrix  is 
positive,  definite,  and  has  condition  number  bounded  by  M. 

10.  Find  the  closest  neighbor  of  P  points  in  K  dimensions. 

11.  Compute  the  first  N  terms  of  the  Qth  composite  of  a  power  series. 

12.  Compute  the  first  N  digits  of  n  (for,  say,  N  =  20,000,000). 
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REDUCIBILITY  AMONG  PROBLEMS.  There  are  many  problems  for  which  the  best  algorithm 
knov.ii  costs  exponential  time.  Such  problems  occur  In  operations  research,  computer 
design,  data  manipulation,  graph  theory  and  mathematical  logic.  Do  there  exist  faster 
algorithms  which  solve  these  problems  in  polynomial  time?  We  don't  know.  What  we  do 
know  is  that  there  is  a  large  class  of  problems  which  are  equivalent  in  that  if  one 
of  them  can  be  solved  in  polynomial  time,  they  all  can. 

For  technical  reasons  this  class  of  problems  is  said  to  be  NP-complete  .  Because 
no  one  has  succeeded  in  devising  a  polynomial  time  algorithm  for  any  of  these  problems, 
many  researchers  believe  that  NP-complete  problems  are  exponentially  hard.  There  is 
no  proof  of  this  and  settling  this  question  is  one  of  the  most  important  open  problems 
in  computational  complexity. 
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ANALYTIC  COMPUTATIONAL  COMPLEXITY.  Many  problems  can  be  only  approximately 
solved.  Examples  are  optimal  estimation,  search  for  an  extremum,  calculation  of 
fixed  points,  and  solution  of  partial  differential  equations.  Indeed,  most  problems 
occuring  in  mathematics,  science,  engineering,  and  economics  can  be  only  approximately 
solved.  On  the  other  hand,  to  lower  the  cost  we  may  choose  to  approximately  solve 
problems  which  could  be  exactly  solved.  Important  examples  are  provided  by  the  app¬ 
roximate  solution  of  NP-complete  problems  and  the  iterative  solution  of  large  sparse 
Umar  systems.  Analytic  Computational  Complexity  is  the  study  of  optimal  algorithms 
tor  problems  which  are  solved  approximately. 

A  central  notion  in  Analytic  Complexity  is  that  of  "information".  The  optimality 
properties  of  a  class  of  algorithms  depend  only  on  the  information  used  by  the  algorithms. 
Tills  permits  an  enormous  simplification  in  the  theory  of  optimal  algorithms. 

Often,  but  not  always,  only  partial  information  is  available.  Partial  information 
means  the  problem  is  not  uniquely  specified  by  the  information.  Partial  information  is 
important  because 

t.  it  is  typical  of  "real  world"  problems 

2.  algorithms  often  utilize  only  partial  information 


We  list  some  of  the  questions  studied  in  analytic  complexity. 

1.  Given  certain  information  and  a  positive  number  is  the  information  "strong" 
enough  to  solve  the  problem  to  within  e  ? 

2.  is  "adaptive  Information"  more  powerful  than  "non-adaptive  information"? 

3.  Given  a  problem,  what  is  the  "optimal  information"  for  its  solution? 

4.  Given  a  problem,  what  Is  the  optimal  algorithm  for  its  solution? 

:.  Given  a  problem,  what  Is  its  complexlty?(ln  analytic  complexity  this  is 

often  known  to  within  very  tight  bounds.) 

h.  Given  two  problems,  which  is  Intrinsically  more  difficult? 
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EVALUATION  OF  THE  SOVIET  ALGORITHM  FOR  LINEAR  PROGRAMMING.  Wo  evaluate  the  new 
algorithm  mentioned  at  the  beginning  of  this  article.  The  following  summarizes  the 
views  of  most  researchers  . 


1.  The  algorithm  is  of  theoretical  interest  since  it  proves  that  linear  programming 
can  be  performed  in  polynomial  time. 

2.  The  algorithm  is  not  currently  competitive  with  the  simplex  method  for  linear 
programming  problems  of  the  size  which  occur  in  practice.  There  is  a  possibility 
that  an  improved  algorithm  would  be  competitive. 

3.  Statements  in  the  New  York  Times  and  the  magazine  Science  re]  ting  linear 
programming  to  the  traveling  salesman  problem  are  incorrect. 

4.  The  new  algorithm  uses  an  iterative  approach  that  might  be  useful  in  solving 
combinatorial  problems  which  are  harder  than  linear  programming. 

5.  The  algorithm  has  been  erroneously  ascribed  solely  to  Khachian.  Suitable 
names  are  "Soviet"  or  "ellipsoidal"  algorithm. 


CONCLUSIONS.  Computational  complexity  deals  with  the  fundamental  issues  of  deter¬ 
mining  the  intrinsic  difficulty  of  mathematically  posed  problems.  Through  the  study 
of  complexity  it  has  been  established  that  certain  problems  are  intrinsically  hard. 
On  the  other  hand,  for  some  problem  areas  new  algorithms  have  been  introduced  which 
arc  far  superior  to  any  previously  known.  Problems  occuring  in  a  rich  diversity  of 
disciplines  will  be  subjected  to  complexity  analysis. 
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